Caching compilation is not alone enough. We must also cache registered tools resolved via #tool directive. Addins (and their dependencies) must also be loaded into app domain / load context in same order as the first time when resolved via #addin directive.

Everything still runs the same as previously. The script itself is still processed by cake to pickup the #addin's and #tool's. Once this is done and the complete script is generated this is where cake would compile the script to run. At this point it checks to see if a valid cached compile scripted exists. If it does then it runs the cached script.

I think this could (should) be true by default. Maybe there can be an additional check to check modification time of files against the build time of the cached assembly to determine whether a new build is required.

They wanted this to be opt-in so that is why it was defaulted to false. There is an additional step to determine if the cached assembly is valid. A hash is taken of the final script and that is used to determine if the cache is valid or not.